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(54) PANORAMA IMAGE GENERATOR AND RECORDING MEDIUM 
(57)Abstract: 

PROBLEM TO BE SOLVED: To provide a panorama image 
generator in which deterioration in the image quality due to 
deviation in the image or double exposure is less even with 
regard to the image including a parallax caused by a 
difference of the depth in the case of photographing when 
movement of a center of a lens of a camera is unavoidable 
for photographing a scene by the camera or the lil<e. 
SOLUTION: A panorama image generator is provided with 
an image input device 101 , a frame memory 102 that stores 
inputted divided images, an image position calculation part 
106 that calculates a synthesis position of the divided 
images, an image parallax extract processing part 107 that 
extracts parallax information between images from the 
image including a parallax in an overlap area between 

adjacent divided images, an intermediate image production processing part 108 that produces 
plural intermediate images from the image including the parallax and an image synthesis 
processing part 1 09 that synthesizes a panorama image from the divided images. The image 
parallax extract processing part 107 corrects a deviation in the images in an overlap area and 
interpolates by the intermediate image produced by the intermediate image production 
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processing part 108 to synthesize the panorama image. 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1, This document has been translated by computer. So the translation may 

not reflect the original precisely. 

2 **** shows the word which can not be translated. 

3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] two or more image pick-up means characterized by providing the 
following -■ or a panorama image hsting device for carrying out 
multiple-times migration of the image pick-up means, making some image 
pick-up images overlap, taking a photograph, connecting an obtained 
division image, and creating a panorama image Said image pick-up means A 
storage means to memorize a division image inputted from said image 
pick-up means An image input-process means to direct the storage place 
address of said captured division image to said storage means while 
directing incorporation of said division image to said image pick-up means 
Parallax in an overlap field between said division images which adjoin an 
image location count means to calculate a synthetic location of said division 
image 

[Claim 2] two or more image pick up means characterized by providing the 
following " or a panorama image Hsting device for carrying out 
multiple-times migration of the image pick-up means, making some image 
pick-up images overlap, taking a photograph, connecting an obtained 
division image, and creating a panorama image Said image pick-up means A 
storage means to memorize a division image inputted from said image 
pick up means An image input-process means to direct the storage place 
address of said captured division image to said storage means while 
directing incorporation of said division image to said image pick-up means 
Parallax in an overlap field between said division images which adjoin an 
image location count means to calculate a synthetic location of said division 
image 
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[Claim 3] It is the panorama image listing device characterized by said 
cementation Une decision means setting a cementation line as an outline 
nearest to an image edge by the side of an overlap field of a photographic 
subject nearest to an image pick up means of an image of said overlap field 
in a panorama image listing device according to claim 2. 
[Claim 4] It is the panorama image listing device characterized by to set said 
cementation line decision means in said overlap field in claim 2 or a 
panorama image listing device according to claim 3, to elect optimal 
cementation line firom two or more cementation lines depended on selection 
of a criteria image at the time of cementation line decision, and combination 
of said cementation line decision method, and to join a division image. 
[Claim 5] Or carry out multiple-times migration of the image pick-up means, 
make some image pick-up images overlap, and a photograph is taken, two or 
more image pick-up means - While being a program for connecting an 
obtained division image and creating a panorama image, directing an image 
input-process means to an image pick-up means and making a division 
image capture A storage means is made to memorize said division image 
which directed the storage place address and was captured to a storage 
means. Make an image location count means calculate a synthetic location of 
said division image, and parallax information between images is made to 
extract from an image which included parallax in an overlap field between 
said adjoining division images in a parallax information extract means. A 
middle image creation means is made to create two or more middle images 
fi-om an image including parallax in said overlap field. While making an 
image composition means compound a panorama image from said division 
image and amending a gap of an image of said overlap field in said parallax 
information extract means in that case A record medium which stored a 
program which performs actuation which interpolates by middle image 
created with said middle image creation means, and compounds a panorama 
image. 

[Claim 6] Or carry out multiple-times migration of the image pick-up means, 
make some image pick-up images overlap, and a photograph is taken, two or 
more image pick-up means - While being a program for connecting an 
obtained division image and creating a panorama image, directing an image 
input-process means to an image pick-up means and making a division 
image capture A storage means is made to memorize said division image 
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which directed the storage place address and was captured to a storage 
means. Make an image location count means calculate a synthetic location of 
said division image, and parallax information between images is made to 
extract from an image which included parallax in an overlap field between 
said adjoining division images in a parallax information extract means. It is 
based on depth information acquired by cementation line decision means 
with said parallax information extract means. A cementation line is made to 
set up on an outline of a photographic subject nearest to an image pick-up 
means of an image of said overlap field. While making an image composition 
means compound a panorama image from said division image and amending 
a gap of an image of said overlap field in said parallax information extract 
means in that case A record medium which stored a program which performs 
actuation which compounds a panorama image by joining a division image 
based on a cementation line called for by said cementation line decision 
means. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] This invention relates to the 
panorama image listing device which compounds a panorama image from 
two or more still digital images. 
[0002] 

[Description of the Prior Art] Two or more digital images which have an 
overlap field mutually connect, two feature particles which are indicated by 
JP,3- 182976, A extract between contiguity images, for example as a method 
of creating a panorama image or a high definition ( high resolution) image, 
and there are a method of connecting an image by making into a cementation 
line the line which connects them, or the method of connecting an image in a 
field in which a concentration difference which is indicated by JP,5-122606,A 
becomes min. Moreover, in case an image is joined, it does not connect with a 
line like a cementation line, but the method of graduating a color tone so that 
concentration change of both images may not become discontinuity in the 
cementation Hne circumference is indicated by the page 466 of an 
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image-analysis handbook of the University of Tokyo Press issue. 
[0003] 

[Problem(s) to be Solved by the Invention] However, in photography of a 
digital image, a photograph may be installed and taken when carrying out 
panphotography and carrying out [ which is depended on a stock ] 
panphotography using a tripod, or so that it may have overlap with two or 
more cameras. In the panphotography by the stock, and photography with 
two or more cameras, as shown in drawing 4 , the lens location of a camera 
moves. Moreover, since the center of rotation and the lens center of a camera 
are not in agreement in many cameras and the combination of a tripod, 
migration of some lenses arises. Although it has been reflected to the 
direction side of a pan on the other hand in the portion from which the depth 
of the photographic subject in an image is different so that drawing 5 may 
see since a difference of these lens locations produces the parallax which 
originates in the difference in the depth of a photographic subject in an 
image pick-up image Also when a portion which has not been reflected to 
another side arising, and the body which moves since a time lag is in 
photography are photoed, the image in overlap is not in agreement. 
[0004] Although the cementation line was joined in the straight Une and the 
straight hne with width of face in the conventional image cementation 
technology, when a difference is among the above images, it cannot 
compound well in many cases. Even if especially the part is connected 
automatically, in the portion which has produced parallax, and the part 
containing a migration body, the gap and the double projection on a 
cementation line arise, and it has become the cause of reducing the quality of 
the joined panorama image remarkably. 

[0005] In case it photos scenery with a camera etc., this invention offers a 
panorama image listing device with httle deterioration of the image quahty 
by gap and double projection of an image also to an image including the 
parallax which the lens center of a camera produces by the difference in 
depth in photography in case migration is not avoided, in order to solve the 
trouble of the above conventional technology. Moreover, deterioration of the 
image quality by gap and double projection of an image offers few panorama 
image listing devices also to the image with which the lens center of a 
camera includes the parallax produced by the difference in depth in 
photography the case where migration is not avoided, and in case including 
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an animal object is not avoided, and an animal object. 
[0006] 

[Means for Solving the Problem] A panorama image listing device concerning 
claim 1 of this invention Or carry out multiple-times migration of the image 
piclc-up means, make some image pick-up images overlap, and a photograph 
is taken, two or more image pick-up means -■ It is a panorama image listing 
device for connecting an obtained division image and creating a panorama 
image. Said image pick-up means. While directing incorporation of said 
division image to a storage means to memorize a division image inputted 
from said image pick-up means, and said image pick-up means, said storage 
means is received. An image input-process means to direct the storage place 
address of said captxired division image, A parallax information extract 
means to extract parallax information between images from an image 
including parallax in an overlap field between said division images which 
adjoin an image location count means to calculate a synthetic location of said 
division image, A middle image creation means to create two or more middle 
images from an image including parallax in said overlap field, While it has 
an image composition means to compound a panorama image from said 
division image and said parallax information extract means amends a gap of 
an image of said overlap field, it is characterized by interpolating by middle 
image created with said middle image creation means, and compounding a 
panorama image. 

[0007] A panorama image listing device concerning claim 2 of this invention 
Or carry out multiple -times migration of the image pick-up means, make 
some image pick-up images overlap, and a photograph is taken, two or more 
image pick-up means It is a panorama image listing device for connecting 
an obtained division image and creating a panorama image. Said image 
pick-up means. While directing incorporation of said division image to a 
storage means to memorize a division image inputted from said image 
pick-up means, and said image pick-up means, said storage means is 
received. An image input-process means to direct the storage place address 
of said captured division image, A parallax information extract means to 
extract parallax information between images from an image including 
parallax in an overlap field between said division images which adjoin an 
image location count means to calculate a synthetic location of said division 
image, A cementation line decision means to set up a cementation line based 
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on depth information acquired by said parallax information extract means on 
an outline of a photographic subject nearest to an image pick-up means of an 
image of said overlap field, While it has an image composition means to 
compound a panorama image fi:om said division image and said parallax 
information extract means amends a gap of an image of said overlap field, 
based on a cementation line called for by said cementation line decision 
means, it is characterized by joining a division image. 
[0008] It is characterized by a panorama image listing device concerning 
claim 3 of this invention setting a cementation line as an outline nearest to 
an image edge by the side of an overlap field of a photographic subject with 
said cementation line decision means nearest to an image pick-up means of 
an image of said overlap field in a panorama image listing device concerning 
claim 2. In the panorama image listing device which the panorama image 
listing device concerning claim 4 of this invention requires for claim 2 or 
claim 3, said cementation line decision means is characterized by to set in 
said overlap field, to elect the optimal cementation line from two or more 
cementation lines depended on selection of a criteria image at the time of 
cementation line decision, and combination of said cementation line decision 
method, and to join a division image. 

[0009] a record medium concerning claim 5 of this invention " two or more 
image pick-up means - or Carry out multiple-times migration of the image 
pick-up means, make some image pick-up images overlap, and a photograph 
is taken. While being a program for connecting an obtained division image 
and creating a panorama image, directing an image input-process means to 
an image pick-up means and making a division image capture A storage 
means is made to memorize said division image which directed the storage 
place address and was captured to a storage means. Make an image location 
count means calculate a synthetic location of said division image, and 
parallax information between images is made to extract from an image 
which included parallax in an overlap field between said adjoining division 
images in a parallax information extract means. A middle image creation 
means is made to create two or more middle images from an image including 
parallax in said overlap field. While making an image composition means 
compound a panorama image from said division image and amending a gap 
of an image of said overlap field in said parallax information extract means 
in that case A program which performs actuation which interpolates by 
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middle image created with said middle image creation means, and 
compounds a panorama image is stored. 

[00 10] a record medium concerning claim 6 of this invention " two or more 
image pick up means " or Carry out multiple-times migration of the image 
pick-up means, make some image pick up images overlap, and a photograph 
is taken. While being a program for connecting an obtained division image 
and creating a panorama image, directing an image input-process means to 
an image pick-up means and making a division image capture A storage 
means is made to memorize said division image which directed the storage 
place address and was captured to a storage means. Make an image location 
count means calculate a synthetic location of said division image, and 
parallax information between images is made to extract from an image 
which included parallax in an overlap field between said adjoining division 
images in a parallax information extract means. It is based on depth 
information acquired by cementation line decision means with said parallax 
information extract means. A cementation line is made to set up on an 
outline of a photographic subject nearest to an image pick up means of an 
image of said overlap field. While making an image composition means 
compound a panorama image from said division image and amending a gap 
of an image of said overlap field in said parallax information extract means 
in that case Based on a cementation line called for by said cementation line 
decision means, a program which performs actuation which compounds a 
panorama image by joining a division image is stored. 
[0011] 

[Embodiment of the Invention] Hereafter, the gestalt of operation of this 
invention is explained, referring to a drawing. Drawing 1 is the block 
diagram showing one example of the operation gestalt concerning claim 1 of 
this invention. 101 ■- a pictxire input device and 102 " a firame memory and 
103 an image output unit and 104 the image input-process section and 
105 " the image pre treatment section and 106 for the middle image 
generation processing section and 109, as for the image output-processing 
section and 111, the image composition processing section and 110 are [ the 
image location count section and 107 / the image parallax extract processing 
section and 108 / the timing-control section and 112 ] RAM. Drawing 3 is 
drawing showing the example of 1 configuration of said RAMI 12. Here, said 
RAMI 12 is a location which stores the information which generated 
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information by each [ OK and ] processing from the predetermined address 
by some processings in the following explanation. 

[0012] Drawing 8 is drawing showing the procedure of processing of the 
operation gestalt concerning claim 1 of this invention. 

[0013] Hereafter, the contents about the operation gestalt which starts claim 
1 of this invention using drawing 1 are explained. 

[0014] If the whole system starts, the timing-control section 111 wiU send a 
reset signal to the image input-process section 104, the image pretreatment 
section 105, the image location count section 106, the image parallax extract 
processing section 107, the middle image generation processing section 108, 
the image composition processing section 109, and the image 
output-processing section 110. 

[0015] The processing in said each block goes into a waiting state until it 
receives the start signal from the timing-control section 111. First, the 
timing-control section 111 goes into a waiting state until it takes out a start 
signal to the image input-process section 104 and receives the terminate 
signal from the image input-process section 104, since an input image is 
stored in a frame memory 102. 

[0016] The image input-process section 104 will perform processing which 
transmits an image to a frame memory 102 to a picture input device 101, if a 
start signal is received from the timing-control section 111. When 
compounding an image on line, a picture input device 101 consists of an 
image input means and an A/D-conversion means for changing into a digital 
picture signal the analog picture signal acquired from an image input means, 
with the signal from the image input-process section 104, is changed into a 
digital picture signal with the A/D-conversion means of the image pick-up of 
an image, and an analog picture signal, and serves as processing transmitted 
to the predetermined address of a frame memory 102. In composition of the 
image in off-line, a picture input device 101 serves as processing which 
transmits the picture signal with which the image input-process section 104 
is memorized by the picture input device 101 to the predetermined address of 
a frame memory 102 in the storage of an image. The image input-process 
section 104 will write image input number of sheets and each storing address 
in the predetermined address of RAMI 12 one by one, if the transfer to a 
frame memory 102 completes all images. Then, the image input-process 
section 104 transmits the terminate signal which notifies termination of 



8/28 



Japanese Publication number : 10-178564 



image input process to the timing-control section 111, and ends processing. 
[0017] If the terminate signal of the transfer to the frame memory of an 
input image is received, the timing-control section 111 transmits a start 
signal to the image pretreatment section 105, and it will go into a waiting 
state until it receives the terminate signal from the image pretreatment 
section 105. 

[0018] If a start signal is received from the timing-control section 111, the 
image pretreatment section 105 wiU read and memorize the image number of 
sheets which should be processed from RAM112, and will set Counter k to 1. 
Next, the address in the frame memory 102 of image I (k) corresponding to 
Counter k is read from RAMI 12, and the predetermined image in a frame 
memory 102 is read from this address (see the drawing 8 (**) and the (**)). 
Processing which removes the known distortion to this image is performed to 
read image I (k). It has the aberration amendment which the lens of for 
example, an image pick-up system has. The image from which it was 
removed is again written out to the same address of a frame memory 102. 
Then, Counter k is increased one and Counter k is compared with image 
number of sheets. The following image will be pretreated if Counter k 
becomes below image number of sheets. If Counter k consists of image 
number of sheets size, the image pretreatment section 105 will transmit the 
terminate signal which notifies termination of image pretreatment to the 
timing-control section 111, and will end processing. 

[0019] If the terminate signal from the image pretreatment section 105 is 
received, the timing-control section 111 transmits a start signal to the image 
location count section 106, and it will go into a waiting state until it receives 
the terminate signal from the image location count section 106. 
[0020] If a start signal is received from the timing-control section 111, the 
image location count section 106 will read and memorize the image number 
of sheets which should be processed from RAMI 12, and wiU set Counter k to 
1. Next, the address in the frame memory 102 of the image corresponding to 
Counter k is read from RAM112, and partial (rxO, ryO) • (rxl, ryl) used as the 
reference field of pattern matching as shown in drawing 9 (a) and drawing 9 
(b) among predetermined image I in a frame memory 102 (k) from this 
address is read. 

[0021] Furthermore, seek-area (sxO, syO) - (sxl, syl) of the image I 
contiguous to the image (k+l) is read similarly. About a seek area, when the 
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physical relationship between images is known to some extent (i.e., when it 
was picturized so that physical relationship might be made regularity, or 
rough physical relationship is given from the outside after the image 
pick up), a seek area is limited and read using the information, and when 
physical relationship is undecided, all the fields of an image are read as a 
seek area. 

[0022] Pattern matching is performed based on the reference field of each 
read image, and a seek area, and the picture signal of a reference field asks 
for the location within a seek area to match, i.e., the location where a 
correlation value with a reference field is the highest, (mxO, myO). A 
correlation value calculates for example, by the bottom formula. 
[0023] 

[Equation l] 



[0024] Here, a brightness value [ in / Rarea and / in R (x y) / the coordinate (x 
y) of a reference field ], a brightness value [ in / in S (x y) / the coordinate (x y) 
of a seek area ], and (offsetx, offsety) are offset firom the zero (0 O) of a 
retrieval image. [ a reference field ] Since the matching position (mxO, myO) 
for which it asked here is a location in each image of the reference field 
started fi-om the original image, and a seek area as shown in drawing 9 (c), it 
amends by the following formulas and calculates the amount of gaps 
between the images in a former image (xmove, ymove). 
[0025] 

[Equation 2] 



[0026] The amount of gaps between images (the amount of parallel 
displacements) is calculable with the above. 

[0027] Moreover, if required, relative angle of rotation etc. will be called for 
according to the direction difference which change of the scale of an image, a 
reference field, and a matching field make from change of the distance 
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between reference fields, and the distance between matching fields by 
preparing two or more reference fields like drawing 9 (d) and drawing 9 (e). 
For example, when the vector passing through the center of the vector Vr 
passing through the center of two or more reference fields and a matching 
field is set to Vm, change of a scale and angle of rotation are called for by the 
following formulas. 
[0028] 

[Equation 3] 



[0029] In this case, that to which only 1/Sk time and - theta (k) rotated Image 
I (k+1) is anew written out from scale S (k) to image I (k), and 
angle -of-rotation theta (k) to the address of the original image I of a frame 
buffer (k+l). The amount of parallel displacements furthermore calculated is 
written in the position of RAMI 12 corresponding to Counter k. After 
performing the above processing, Counter k is increased one and the 
comparison with Counter k and image number of sheets is performed. From 
image number of sheets, if Counter k is smallness, it will perform 
computation of the image location of a degree. Whether Counter's k being 
equal to image number of sheets and the terminate signal to which the image 
location count section 106 will notify termination of image location 
computation to the timing-control section 111 if it becomes size are 
transmitted, and processing is ended. 

[0030] If the terminate signal firom the image location count section 106 is 
received, the timing-control section 111 transmits a start signal to the image 
parallax extract section 107, and it will go into a waiting state until it 
receives the terminate signal fi-om the image parallax extract processing 
section 107. 

[0031] If a start signal is received fi-om the timing-control section 111, the 
image parallax extract processing section 107 will read and memorize the 
image number of sheets which should be processed fi-om RAMI 12, and will 
set Counter k to 1. Next, the address in the frame memory 102 of the image 
corresponding to Counter k is read from RAMI 12, and the image in a frame 
memory 102 is read from this address. This is called image I (k). 
Furthermore, the image (it corresponds to a counter k+l) contiguous to the 
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image is read. This image is called Image I (k+l). Then, the amount of 
parallel displacements calculated in the image location count section 106 is 
read from RAM112, and it asks for the field where Image I (k+l) overlaps 
image I (k) about image I (k) and Image I (k+l), respectively. 
[0032] And the overlap portion of image I (k) is divided, each is made into a 
reference field, and the location which corresponds from Image I (k+l) is 
searched using the technique of pattern matching. Under the present 
circumstances, what is necessary is just to perform retrieval only from a 
longitudinal direction, since the gap between the images by parallax 
searches. This amount of gaps is the amount of motions on the appearance of 
the body by parallax, and since it originates in depth, it can presume the 
depth of the body in the direction of a gap, and the reference field from the 
amount of gaps (see drawing 10 ). 

[0033] Moreover, on the relation of parallax, when the point of the distance of 
a certain depth is made in agreement, the amount of gaps to right-hand side 
becomes large also within the overlap of the same width of face, so that the 
body nearer to [ point / a certain ] a camera has the larger amount of gaps to 
left-hand side and it is further than a certain point from a camera. Here, the 
point of a certain distance may add amendment based on the amount of gaps 
in the overlap for which it asked here, although it becomes the distance 
corresponding to the amount of parallel displacements calculated in the 
image location count section 106. For example, if it will shift and a reference 
value will be made into the minimum value of an amount, if it doubles 
among an image in the location where depth is the deepest, and it doubles 
among an image in the location where depth is the shallowest, it shifts, and a 
reference value is made into the maximum of an amount, and it is good also 
considering the average of the amount of gaps as a reference value. A 
reference value and a difference are taken and each amount of gaps in the 
overlap field for which applied the amount of parallel displacements in 
RAMI 12 by the reference value for which it asked, and it asked now is 
amended. By doing so, the amount of gaps made into the reference value can 
amend to 0. 

[0034] In this processing, it does not ask for exact depth but the amount of 
gaps between the parallax accompanying the difference of depth, i.e., a pixel, 
itself is stored in RAMI 12 for every field and every pixel. 
[0035] Drawing 6 is drawing having shown the situation of matching for 
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performing a parallax extract. 

[0036] The round mark in a field shows a field center in the reference field 
where the grid of an image (A) was divided. The frame in an image (B) shows 
an overlap field, and the arrow head which a round mark corresponds to the 
round mark which shows the reference field center of an image (A), and is 
extended fi-om there indicates the amount of gaps to be the direction of the 
gap by matching. Here, if the image was photoed like drawing 6 in the 
location on the right of the location where the right-hand side image photoed 
the left-hand side image, it can say that it is shallower in depth as a gap 
rightward is large, depth is deep and a gap leftward has the large amount of 
gaps. The left was just taken and the image (C) showed this amount of gaps 
as a brightness value of a pixel. It is shown that a darker portion has smaller 
parallax, that is, depth is deep, and a brighter portion has larger parallax, 
that is, depth is shallow. Moreover, a finer parallax extract is attained by 
making a reference field still smaller (see drawing 8 (Ha) and the (**)). 
[0037] Similarly, a depth extract is performed by making image I (k) into a 
retrieval side, making Image I (k+l) as a reference side. The parallax extract 
result of Image I (k+l) is similarly stored in RAM112. 
[0038] If image I (k) and depth extract processing of Image I (k+l) finish. 
Counter k is increased one and Counter k is compared with image number of 
sheets. From image number of sheets, if Counter k is smallness, it will 
perform image I (k) and depth extract processing to Image I (k+l). If Counter 
k becomes more than image number of sheets, the image parallax extract 
processing section 107 will transmit the terminate signal which notifies 
termination of image parallax extract processing to the timing-control 
section 111, and will end processing. 

[0039] If the terminate signal fi-om the image parallax extract processing 
section 107 is received, the timing- control section 111 transmits a start 
signal to the middle image generation section 108, and it will go into a 
waiting state until it receives the terminate signal from the middle image 
generation section 108. 

[0040] If a start signal is received from the timing-control section 111, the 
middle image generation processing section 108 will read and memorize the 
image number of sheets which should be processed from RAM112, and will 
set Counter k to 1. Next, the address in the frame memory 102 of the image 
corresponding to Counter k is read from RAMI 12, and image [ in a frame 
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memory 102 ] I (k) is read from this address. Furthermore, the image (it 
corresponds to a counter k+l) (k+l) I contiguous to the image is read. Then, 
the amount of parallel displacements calculated in the image location count 
section 106 is read from RAM112, and it asks for the overlap field of image I 
(k), and the overlap field of Image I (k+l), respectively. 
[0041] A middle image (Intermidiate Image) is an image created by count in 
the image picturized in the location which is the middle of both the view from 
the image of two sheets picturized from the location where a view is different. 
Let the location which picturized image I (k) be the image pick-up location 
Tk. the middle image Ilk (1) picturized from 2:n-2, -, the image pick-up 
location divided interiorly to n-i:i from the location which divides the image 
in overlap into n-1 horizontally in a field, and divides interiorly the image 
pick-up location Tk and image pick-up location Tk-»-l from the left to i:n-l 
here, respectively, and 1= " 1, 2, n-1 are created by the following methods. 
[0042] The creation method of the middle image in middle image generation 
in the image pick-up location Tk and the location which divides interiorly 
image pick up location Tk-<-l to 1-n-l is shown. The amount of gap gaps 
between the images of image I (k) and Image I (k-H) for which it asked in the 
parallax extract processing section 107 is read from RAMI 12 for every pixel 
or field. 

[0043] As shown in drawing 7 (A) and drawing 7 (B), the point of the 
arbitration in overlap field Rl(rxO, ryO)- (rxl, ryl) with the image I (k-H) of 
image I (k) is set to PI. D (Pi) and the amount of gaps in a point PI are set 
[ the coordinate of a point PI ] to M (Pi) (arrow head of drawing 7 (A)) for (xl, 
yl), and the brightness value in a point PI. The point of the arbitration in 
overlap field R2(rx2, ry2)- (rx3, ry3) with image [ of Image I (k-i-l) ] I (k) is set 
to P2, and D (P2) and the amount of gaps in a point P2 are set [ the 
coordinate of a point P2 ] to M (P2) (arrow head of drawing 7 (B)) for (x2, y2), 
and the brightness value 

[0044] Middle image Ilk (D drawing 7 (C) on the basis of image I (k) is 
created first. The brightness value of the point (x-t-M(Pl) *l/n-rxO, yryO) on 
the middle image Ilk (1) is set to D (Pi) to each point PI (x y) in the overlap 
field Rl, rxO <=x<=rxl, and ryO <=y<=ryl. 

[0045] When the middle image Ilk (1) is created by image I (k), in a portion 
with a middle image, the brightness value of two or more pixels may be 
written in by the difference in the amount of gaps by the difference in the 
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parallax for every pixel, or an undefined pixel like the black pixel portion of 
drawing 7 (C) may arise by it. During generation of a middle image, when 
the brightness value is already written in, priority is given to the brightness 
value of a pixel with the large amount M of gaps (Pi). The more the amount 
M of gaps (Pi) is large, it is a body near a camera and, the more is because it 
is unnatural that the brightness value of a back body or a background comes 
from it. 

[0046] Moreover, about an undefined pixel, it is interpolated by middle image 
IIk+l(l) created fi-om Image I (k+l) as follows. 

[0047] Middle image Ilk' (1) drawing 7 (D) on the basis of Image I (k+l) is 
created like creation of the middle image Ilk (1). 

[0048] The brightness value of the point (x+M(P2) *(n-l)/n-rx0, yryO) on 
middle image Ilk' (1) is set to D (P2) to each point P2 (x y) in the overlap field 
R2, rx2 <=x<=rx3, and ry2 <=y<=ry3. However, priority is given to the 
brightness value of a pixel with the large amount M of gaps (Pi) when the 
brightness value is already written in, as stated previously. Then, it 
interpolates by the pixel of corresponding Ilk' (1) using Ilk' (1) created fi-om 
Image I (k+l) and the amount of gaps for every pixel of the about the 
undefined pixel in Ilk (1) created from image I (k) and the amount of gaps for 
every pixel of the. 

[0049] When the part where the brightness value is not written in a middle 
image finally exists, it interpolates with the brightness value in which the 
perimeter is written. Drawing 14 shows the situation of interpolation. That 
by which the round mark went [ the rectangular head of each ] into the 
center in one pixel is the pixel of the undefined. Now, the brightness value of 
the undefined pixel surrounded by the thick wire is considered as the 
average which is the pixel by which the triangular mark went into the center 
in the pixel already defined among about [ which were surrounded with the 
thick dashed line ] eight, and drawing 14 . The above actuation is repeated 
about all undefined pixels. After interpolation of other undefined pixels 
finishes, interpolation actuation is performed again, and if all of about eight 
pixel are undefined, it will repeat until an undefined pixel is lost. 
[0050] Moreover, if interpolation is performed from an undefined pixel with 
many about eight pixels already defined, a more quahty interpolation image 
will be obtained. Although the middle image Ilk (1) is generated above, the 
field used for composition is (dx* (M), 0)- (dx*l, (ryl-ryO)). Therefore, only 
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the portion is written in a frame memory 102, and middle image generation 
of Tn-l in the overlap field of image I (k) and Image I (k+l) is finished. 
[0051] 1= - the number of sheets of the middle image created to RAM112 
when performing said middle image creation processing to 1, 9 - the 
predetermined address ■■ writing in - Counter k - the increase of one - it 
carries out and Counter k is compared with image number of sheets. From 
image number of sheets, if Counter k is smallness, it will perform said 
middle image processing to the image equivalent to the updated counter. If 
Counter k is equal to image number of sheets, the middle image generation 
processing section 108 wiU transmit the terminate signal which notifies 
termination of middle image generation processing to the timing-control 
section 111, and will end processing. 

[0052] If the terminate signal from the middle image generation processing 
section 109 is received, the timing-control section 111 transmits a start 
signal to the image composition processing section 109, and it will go into a 
waiting state until it receives the terminate signal from the image 
composition processing section 109. 

[0053] If a start signal is received from the timing-control section 111, the 
image composition processing section 109 will read and memorize the image 
number of sheets which should be processed from RAMI 12, and wiU set 
Counter k to 1. Next, the address in the frame memory 102 of the image 
corresponding to Counter k is read from RAM112, and image [ in a frame 
memory 102 ] I (k) is read from this address. 

[0054] When Counter k is 1, image I (k) is written in the predetermined 
location of the output image field of a frame memory 102. 
[0055] Counter k calls amount [ with Image I (k-l) ] of gaps M (k) from 
RAM112, in being larger than 1. Overlap field Rk(rxO, ryO)- (rxl, ryl) with 
Image I (k-l) is calculated. Moreover, the middle image number of sheets n 
to the image I (k-l) created in the middle image generation processing 
section 108 and image I (k) is called from RAMI 12. A location Pi is [0056] 
when the attachment location of n is set to Pi, the middle image Ilk (i), i= 1, 

[Equation 4] 
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[0057] It can come out and calculate and the middle image Ilk (i), i= 1, n 
are copied to the synthetic image memory location Pi. Next, the portion after 
the right of the x coordinate rxl of image I (k) is copied after a location 
(sigmaki= 1 M (i) + rxl, ryl) (see the drawing 8 (**) and the (**)). 
[0058] If the above processing finishes, Counter k is increased one and 
Counter k is compared with image number of sheets. If Counter k becomes 
below image number of sheets, said processing to updated image I (k) will be 
continued, and wiU be performed. If Counter k consists of image number of 
sheets size, the image composition processing section 109 will transmit the 
terminate signal which notifies termination of image composition processing 
to the timing-control section 111, and will end processing. 
[0059] If the terminate signal from the image composition processing section 
109 is received, the timing-control section 111 transmits a start signal to the 
image output-processing section 110, and it will go into a waiting state until 
it receives the terminate signal from the image output-processing section 110. 
The image output-processing section 110 will read the output image address 
in a frame memory 102 from RAM112, if the start signal firom the 
timing-control section 111 is received. Next, delivery and an output picture 
signal are read from a frame memory 102 one by one, and the start signal for 
starting an image output to the image output unit 103 is outputted to the 
image output unit 103. If it finishes outputting an output image fi-om the 
image output unit 103, the image output-processing section 110 will 
transmit the terminate signal which notifies termination of image output 
processing to the timing-control section 111, and will end processing. 
[0060] The timing-control section 111 will end all, if the terminate signal 
fi:om the image output-processing section 110 is received. 
[0061] The difference in the parallax between the photographic subjects in 
two overlap of an image is mitigated by creation of a middle image by the 
flow of a series of above processings, and shifting, or becoming a duplex and 
being compounded of the done synthetic image Uke before, also in the portion 
in which the difference in parallax existed, is lost, A more natural panorama 
image is obtained by making [ many ] the number of creation of a middle 
image, i.e., the number of partitions in overlap, with a natural thing. 
Moreover, in the operation gestalt concerning claim 1 of this invention, in 
order to compound the photographic subject according to which it shifts with 
parallax and with which an amount is different by the overlap of the same 
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width of face, as for the thing near a camera, width of face is compounded 
more narrowly, and a far thing is compounded broadly. What is necessary is 
just to amend the whole amount of gaps in the image parallax extract 
processing section 107 by making the amount of gaps of the body near a 
camera into a reference value to make the portion near a camera natural. 
Moreover, what is necessary is just to amend the whole amount of gaps in 
the image parallax extract processing section 107 by making the amount of 
gaps of a body far from a camera into a reference value to make a portion far 
&om a camera natural. 

[0062] As mentioned above, the example given here is an example of the 
operation gestalt concerning claim 1 of this invention, and unless the main 
point of this invention is changed, it is not limited to the contents of this 
example. 

[0063] Drawing 2 is the block diagram showing one example of the operation 
gestalt concerning claim 2 of this invention. 

[0064] 101 -- a picture input device and 102 - a frame memory and 103 - an 
image output unit and 104 the image input-process section and 105 - the 
image pre treatment section and 106 ■* for the cementation Une decision 
processing section and 109', as for the image output-processing section and 
111, the image composition processing section and 110 are [ the image 
location count section and 107' / the image parallax extract processing 
section and 108' / the timing-control section and 112 ] RAM. 
[0065] Drawing 11 is drawing showing the procedure of processing of the 
operation gestalt concerning claim 2 of this invention. 

[0066] Hereafter, the contents about the operation gestalt which starts claim 
2 of this invention using drawing 2 are explained. 

[0067] Since the same processing as the operation gestalt concerning claim 1 
of this invention is performed, each function of the image input-process 
section 104, the image pretreatment section 105, and the image location 
count section 106 and the processing to the image location count section 106 
are omitted for details. 

[0068] If the terminate signal from the image location count section 106 is 
received, the timing-control section 111 transmits a start signal to image 
parallax extract processing section 107', and it will go into a waiting state 
until it receives the terminate signal from image parallax extract processing 
section 107*. 
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[0069] If a start signal is received from the timing-control section 111, image 
parallax extract processing section 107' will read and memorize the image 
number of sheets which should be processed from RAM112, and will set 
Counter k to 1. Next, the address in the frame memory 102 of the image 
corresponding to Counter k is read from RAMI 12, and image [ in a frame 
memory 102 ] I (k) is read from this address. Furthermore, the image (it 
corresponds to a counter k+l) (k+l) I contiguous to the image is read. Then, 
the amount of parallel displacements calcidated in the image location count 
section 106 is read from RAM112, and it asks for the overlap field of image I 
(k), and the overlap field of Image I (k+l), respectively. 

[0070] And the overlap portion of image I (k) is divided, each is made into a 
reference field, and the location which corresponds from Image I (k+l) is 
searched using the technique of pattern matching. Under the present 
circumstances, what is necessary is just to perform retrieval only from a 
longitudinal direction, since the gap between the images by parallax 
searches. This amount of gaps is the amount of motions on the appearance of 
the body by parallax, and since it originates in depth, it can presume the 
depth of the body in the direction of a gap, and the reference field from the 
amount of gaps. Moreover, on the relation of parallax, when the point of a 
certain distance is made in agreement, the amount of gaps to right-hand side 
becomes large also within the overlap of the same width of face, so that the 
body nearer to [ point / a certain ] a camera has the larger amount of gaps to 
left-hand side and it is further than a certain point from a camera. 
[0071] When it asks for what has depth most, that is, the left is made 
positive here in the field for which it asked, or a pixel, a field with the 
smallest amount of gaps and the amount of gaps in there subtract from the 
distance between each field or a pixel, and the portion which has depth most 
inside an image in a gap of an image considers as a gap of criteria by adding 
to the amount of parallel displacements calculated in the image location 
count section 106. In connection with it, the amount of parallel 
displacements in RAMI 12 and each amount of gaps in an overlap field are 
also amended. Similarly, a parallax extract is performed by making image I 
(k) into a retrieval side, making Image I (k+l) as a reference side. The 
parallax extract result of Image I (k+l) is similarly stored in RAM112. 
[0072] If image I (k) and parallax extract processing of Image I (k+l) finish. 
Counter k is increased one and Counter k is compared with image number of 



19/28 



Japanese Publication number : 10-178564 



sheets. From image number of sheets, if Counter k is smallness, it will 
perform image I (k) and parallax extract processing to Image I (k+l). If 
Counter k becomes more than image number of sheets, image parallax 
extract processing section 107' will transmit the terminate signal which 
notifies termination of image parallax extract processing to the 
timing-control section 111, and will end processing. 

[0073] If the terminate signal fi-om image parallax extract processing section 
107' is received, the timing-control section 111 transmits a start signal to 
cementation Une decision processing section 108', and it will go into a 
waiting state until it receives the terminate signal from cementation line 
decision processing section 108'. 

[0074] If a start signal is received firom the timing-control section 111, 
cementation hne decision processing section 108' will read and memorize the 
image number of sheets which should be processed fi:om RAMI 12, and will 
set Counter k to 1. Next, the address in the frame memory 102 of the image 
corresponding to Counter k is read from RAM112, and image [ in a frame 
memory 102 ] I (k) is read from this address. Furthermore, the image (it 
corresponds to a counter k+l) (k+l) I contiguous to the image is read. Then, 
image I (k) for which it asked by image parallax extract processing section 
107', and the amount of parallax of Image I (k+l) are read from RAM112. 
[0075] Here explains respectively how to ask for the cementation line from 
image I (k) in claims 2, 3, and 4, using drawing 11 , and 12 and 13. image 
parallax extract processing section 107' - image I (k) " (" the point, i.e., the 
point of filUng a bottom type, of changing from the parallax image ( drawing 
11 (c)) to image I (k) for which it asked from drawing 11 (a)) and Image I 
(k+l) ( drawing 11 (b)) so that the parallax of an image may become larger 
seen from an image right end is extracted. 
[0076] 

[Equation 5] 

[0077] Here, the amount of parallax [ in / in E (x y) / the coordinate (x y) of an 
image ] and Threshold are suitable thresholds. The white round mark in the 
image in drawing 11 (e) is hit. These sequence of points will show the border 
line on the right-hand side of a photographic subject, its portion whose image 
I (k) is not visible can be seen on the left-hand side of the location which 
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corresponds on Image I (k+l), and its difference between image I (k) (k+l) is 
especially large on the left-hand side near [ these ] a point. Moreover, in near 
the right of these points, since the same thing can be seen with both images, 
the difference of image I (k) and I (k+l) will be small. 

[0078] The extract method of the cementation line in claim 2 is the method of 
extracting a point with as same the parallax as possible from the large 
sequence of points of parallax change for which it asked in consideration of 
the above, that is, extracting the border line of near and the same body to a 
camera, and making it a cementation line. Moreover, when there is no large 
part of those parallax change up and down like drawing 11 (e), let the line 
taken down from the edge of the border line nearest to them to the upper 
limit or lower limit of an image be a cementation line. 

[0079] The extract methods of the cementation Une in claim 3 are the large 
sequence of points of parallax change for which it asked similarly, and are 
the sequence of points exceeding the existing threshold Threshold in an 
overlap field nearest to the image edge by the side of an overlap field, i.e., the 
method of setting to cementation line L (k) the line which connects the 
sequence of points of most right-hand side here, (see drawing 12 (i)). 
[0080] The extract method of the cementation line in claim 4 is as follows. 
The extract method of the cementation line in said claim 2 and claim 3 can be 
adapted similarly by asking for the large sequence of points of the parallax 
change from the left on the right on the basis of Image I (k+l). That is, the 
candidate of two or more cementation lines is called for by asking for a 
cementation line from image I (k) and Image I (k+l) by two methods. The 
candidate of the cementation line of further the plurality in changing the 
value of a threshold Threshold is obtained, them - Li (k) and i= - it 
considers as 1, 2, and - (see drawing 13 (I) and the (II)). 
[0081] the cementation line candidate Li (k) and i= - what [ thing ] is the 
optimal at 1, 2, and - serves as a cementation line by which the following 
values become the smallest. 
[0082] 

[Equation 6] 



[0083] Here, C is a brightness value (or) in the point (x y) on the cementation 
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line [ in / it is a path on the cementation Hne Li (k), and / in Li (k), and (x, y) / 
image I (k) ] Li (k). RGB each a value - a vector - Li - ' (- k -) " (-- x 
I y t g^j^ image *■ I (k+l) it can set - cementation - a line - Li - (-- k 
--) " a top " an image - I - (- k -) - it can set - a point (x y) corresponding 
- a point (x', yO - a top " brightness It asks for the cementation line 
candidate Li (k) who makes a top type min, and let it be a cementation line 
(see drawing 13 (III))- Cementation line L (k) for which it asked as 
mentioned above is written out to the predetermined address of RAMI 12 one 
by one, and it ends. 

[0084] If image I (k) and cementation line decision processing of Image I 
(k+l) finish, Counter k is increased one and Counter k is compared with 
image number of sheets. From image number of sheets, if Counter k is 
smallness, it will perform image I (k) and cementation line decision 
processing to Image I (k+l). If Counter k becomes more than image number 
of sheets, cementation line decision processing section 108' will transmit the 
terminate signal which notifies termination of cementation line decision 
processing to the timing-control section 111, and will end processing. 
[0085] If the terminate signal from cementation Une decision processing 
section 108* is received, the timing-control section 111 transmits a start 
signal to image composition processing section 109', and it will go into a 
waiting state until it receives the terminate signal from image composition 
processing section 109'. 

[0086] If a start signal is received from the timing-control section 111, image 
composition processing section 109' will read and memorize the image 
number of sheets which should be processed from RAMI 12, and will set 
Counter k to 1. Next, the address in the frame memory 102 of the image 
corresponding to Counter k is read from RAM112, and image [ in a frame 
memory 102 ] I (k) is read from this address. When Counter k is 1, image I 
(k) is written in the predetermined location of the output image field of a 
frame memory 102. 

[0087] Counter k calls amount [ with Image I (k-l) ] of parallel displacements 
M (k), and the cementation line L (k-l) from RAMI 12, in being larger than 1. 
[0088] The location which sets image I (k) is [0089]. 
[Equation 7] 
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[0090] It is alike and is obtained more. Moreover, a cementation line is also 
amended in the location which shifted only Pk-1. 

[0091] In case image I (k) is copied to synthetic memory, it carries out on 

condition that the following and an image is compounded. 

[0092] ■ Don't write in the pixel which becomes on the left of cementation line 

Lk-1. 

■ a part for w dots of right of cementation hne Lk-1 - concentration - 
therefore, perform smoothing with image I (k) flat and smooth. 
- Write in a pixel as it is after w dots of cementation Une Lk-right of 1. 
[0093] Here, w is a suitable constant value. By performing concentration 
**** by these w dots, cementation becomes nature more. In drawing 11 , a 
mask will be covered like drawing 11 (h), the field which performs 
concentration smoothing will be prepared, and composition with drawing 11 
(g) will generate a synthetic image hke drawing 11 (i). 
[0094] If the above processing finishes, Counter k is increased one and 
Counter k is compared with image number of sheets. If Counter k becomes 
below image number of sheets, said processing to updated image I (k) will be 
continued, and will be performed. If Counter k consists of image number of 
sheets size, image composition processing section 109' will transmit the 
terminate signal which notifies termination of image composition processing 
to the timing-control section 111, and will end processing. 
[0095] If the terminate signal fi-om image composition processing section 109' 
is received, the timing-control section 111 transmits a start signal to the 
image output-processing section 110, and it will go into a waiting state until 
it receives the terminate signal from the image output-processing section 110. 
[0096] The image output-processing section 110 will read the output image 
address in a fi-ame memory 102 firom RAMI 12, if the start signal from the 
timing-control section 111 is received. Next, delivery and an output picture 
signal are read firom a frame memoiy 102 one by one, and the start signal for 
starting an image output to the image output unit 103 is outputted to the 
image output unit 103. If it finishes outputting an output image from the 
image output unit 103, the image output-processing section 110 will 
transmit the terminate signal which notifies termination of image output 
processing to the timing-control section 111, and will end processing. 
[0097] The timing-control section 111 ends all by receiving the terminate 
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signal from the image output-processing section 110. 
[0098] Joining in the portion from which parallax is greatly different by 
forming a cementation line on the outline in the photographic subject near a 
camera although it becomes the cause which will become a gap and a duplex 
if a photographic subject with the difference of the parallax in the overlap 
field of the image of two sheets nearer to a camera is larger by the flow of 
processing of the above single string and composition is performed there 
decreases, it becomes more natural [ a cementation image ] than before, and 
quality improves. 

[0099] Moreover, with the operation gestalt concerning claims 2, 3, and 4 of 
this invention, even if it is the image which includes an animal object in an 
image, since it appears as parallax, in a parallax extract, cementation 
becomes possible similarly at high quaUty. As mentioned above, the example 
given here is an example of the example concerning claims 2, 3, and 4 of this 
invention, and unless the main point of this invention is changed, it is not 
limited to the contents of the example. 

[0100] Moreover, in the panorama image listing device of this invention, the 
programs which realize processing which creates a series of panorama 
images are the contents of the following stored in record media, such as 
media, such as a floppy disk, a hard disk, a magnetic tape, CD-ROM / optical 
disk / magneto-optic disk, and MD, and ROM/RAM memory. 
[0101] as an example of the contents of a program stored - two or more 
image pick-up means ■■ or While making it incorporate by carrying out 
multiple-times migration of the image pick-up means, making some image 
pick-up images overlap, and an image input-process means directing the 
division image photoed and obtained to an image pick-up means A storage 
means is made to memorize said division image which directed the storage 
place address and was captxired to the storage means. Make an image 
location count means calculate the synthetic location of said division image, 
and the parallax information between images is made to extract from the 
image which included the parallax in the overlap field between said 
adjoining division images in the parallax information extract means. A 
middle image creation means is made to create two or more middle images 
from an image including the parallax in said overlap field. While making an 
image composition means compound a panorama image from said division 
image and amending a gap of the image of said overlap field in said parallax 
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information extract means in that case Actuation which interpolates by the 
middle image created with said middle image creation means, and 
compounds a panorama image is performed. 

[0102] moreover, as another example of the contents of a program stored two 
or more image pick-up means - or while making it incorporate by directing 
to an image pick-up means, an image input-process means the division 
image which carried out multiple-times migration of the image pick-up 
means, and some image pick-up images were made to overlap, took a 
photograph, and was obtained A storage means is made to memorize said 
division image which directed the storage place address and was captured to 
the storage means. Make an image location count means calculate the 
synthetic location of said division image, and the parallax information 
between images is made to extract from the image which included the 
parallax in the overlap field between said adjoining division images in the 
parallax information extract means. It is based on the depth information 
acquired by the cementation line decision means with said parallax 
information extract means. A cementation line is made to set up on the 
outline of the photographic subject nearest to the image pick-up means of the 
image of said overlap field. While making an image composition means 
compound a panorama image from said division image and amending a gap 
of the image of said overlap field in said parallax information extract means 
in that case Based on the cementation line called for by said cementation line 
decision means, actuation which compounds a panorama image by joining a 
division image is performed. 

[0103] In addition, the example of contents given here is an example of the 
example concerning claims 5 and 6 of this invention, and unless the main 
point of this invention is changed, it is not limited to the above-mentioned 
contents. 

[0104] As mentioned above, the program in this invention is using the 
above-mentioned record medium, and can be used also in various 
information machines and equipment, such as a personal computer, a 
Personal Digital Assistant, and a camera one apparatus image processing 
system. 
[0105] 

[Effect of the Invention] By the panorama image fisting device of this 
invention, the following effects are acquired in each claim as mentioned 
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above. 

[0106] When there is parallax by the depth of a photographic subject, in 
claim 1 and claim 5 of this invention, the effect which can generate a 
panorama image natural than before is acquired by searching for the 
parallax resulting from the depth of a photographic subject by count within 
overlap between images, and interpolating the image in overlap using the 
middle image which interpolates the location where the image was taken 
based on it. 

[0107] The paraUax which originates in the depth of a photographic subject 
in claim 2 and claim 6 of this invention, By searching for an animal object by 
count within overlap between images, and using Rhine of an image with 
which the difference of the depth of the body between both images does not 
include a large location based on it as cementation Rhine When there is 
parallax by the depth of a photographic subject, or when it includes an 
animal object, it is more nearly quality than before and the effect which can 
generate a natural panorama image with Httle distortion is acquired. 
[0108] In claim 3 of this invention, when a nearby photographic subject is a 
complicated configuration, even when a cementation line is not called for 
well, by invention of said claim 2, it is forming a cementation hne with the 
outline of a set of a comparatively near photographic subject, and it is quality 
and the effect which can generate a natural panorama image with little 
distortion is acquired. 

[0109] In claim 4 of this invention depending on the location of the 
photographic subject in overlap The selection of an image which may not 
work and is made into criteria in that case by invention of said claim 2 and 
claim 3 if the image made into criteria is always fixed in the one direction, By 
selecting the candidate of two or more cementation lines created by the 
combination of the cementation line decision method in invention of said 
claim 2 and claim 3, and selecting from them the optimal cementation line by 
which the difference between the images on a cementation Une becomes the 
smallest It is more quality for stabihty and the effect which can generate a 
nat\iral panorama image with little distortion is acquired. 
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[Brief Description of the Drawings] 

[Drawing ll It is a block diagram explaining one example of the operation 
gestalt in claim 1 of this invention. 

[Drawing 2] It is a block diagram explaining one example of the operation 
gestalt in claim 2 of this invention. 

[Drawing 3] It is explanatory drawing having shown the example of 1 
configuration of RAM in the operation gestalt in claim 1 of this invention. 
[Drawing 4] It is explanatory drawing having shown the situation of 
migration of a camera or the image photography which gave the overlap field 
with two or more cameras. 

[Drawing 5] It is the image photoed by drawing 4 , and is explanatory 
drawing having shown that the image in overlap was not in agreement with 
the difference in depth. 

[Drawing 6] It is explanatory drawing having shown the situation of 
matching for performing a parallax extract. 

[Drawing 7] It is explanatory drawing having shown the situation of creation 
of a middle image. 

[Drawing 8] It is explanatory drawing having shown the procedure of 

processing of the operation gestalt in claim 1 of this invention. 

[Drawing 9] It is explanatory drawing having shown signs that it asked for 

an image location in claim 1 and claim 2 of this invention. 

[Drawing lOl It is explanatory drawing having shown the relation between 

the amount of gaps of matching between images, and the depth from a 

camera. 

[Drawing 111 It is explanatory drawing having shown the procedxire of 

processing of the operation gestalt in claim 2 of this invention. 

[Drawing 12] It is explanatory drawing having shown the situation of the 

decision of the cementation line in claim 3 of this invention. 

[Drawing 13] It is explanatory drawing having shown the situation of the 

decision of the cementation line in claim 4 of this invention. 

[Drawing 14] It is explanatory drawing having shown the situation of the 

interpolation method from the perimeter of an undefined pixel. 

[Description of Notations] 

101 Picture Input Device 

102 Frame Memory 

103 Image Output Unit 
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104 Image Input-Process Section 

105 Image Pretreatment Section 

106 Image Location Count Section 

107 Image Parallax Extract Processing Section 

108 Middle Image Generation Processing Section 

109 Image Composition Processing Section 

110 Image Output-Processing Section 

111 Timing- Control Section 

112 RAM 

107' Image parallax extract processing section 
108' Cementation line decision processing section 
109' Image composition processing section 
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